[FEAT]: Add durable terminal evaluation contract by spencrr · Pull Request #146 · microsoft/RAMPART

Spencer Schoenberg (spencrr) · 2026-08-04T00:41:49Z

Description

Adds the per-execution provenance needed before final-trace verdict cadence changes. Result.terminal_evaluation stores the evaluator output for the terminal trace, Result.trace_end_reason records why trace production ended, and Turn.eval_purpose identifies online stop checks. Result.turn_evaluations makes the online evidence boundary explicit while eval_results remains a compatibility view.

The direct attack and probe resolvers require one evaluation and reject unknown runtime outcomes instead of falling through. Population provenance now shares validation across PopulationRef, PopulationResult, and execute_trials_async, with invalid thresholds rejected before an execution factory runs.

This PR also includes the persistence work previously split into #147 so the contract cannot land without transport support. The current xdist v2 envelope and JSON report carry terminal evaluation, trace-end reason, turn purpose, and population provenance together. Malformed worker data fails closed, including overflowing confidence values, and oversized results produce bounded incomplete markers while retaining population provenance when it fits.

Breaking changes

None for valid callers. Invalid population provenance and malformed evaluator outcomes now fail early instead of being accepted or falling through.

Checklist

pre-commit run --all-files passes
Tests cover contract fields, compatibility views, resolvers, population validation, xdist transport, truncation, JSON output, malformed confidence, and real worker/controller persistence
Documentation updated

Validation: 1,122 unit tests pass. Strict documentation build and all pre-commit checks pass.

azure-pipelines · 2026-08-04T00:41:59Z

Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run.

Copilot

🟡 Changes recommended

A huge terminal confidence value can raise an uncaught overflow and abort controller-side xdist merging.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds durable terminal-evaluation and population provenance across core results, reporting, and xdist transport.

Changes:

Adds terminal evaluation, trace-end reason, evaluation-purpose, and resolver APIs.
Centralizes population validation.
Extends persistence, truncation handling, tests, and documentation.

File summaries

File	Description
`rampart/core/_population.py`	Adds shared population validation.
`rampart/core/types.py`	Adds provenance enums and turn purpose.
`rampart/core/result.py`	Extends results and verdict resolution.
`rampart/core/execution.py`	Validates trial parameters early.
`rampart/core/__init__.py`	Exports new core APIs.
`rampart/pytest_plugin/_xdist.py`	Transports and validates provenance.
`rampart/reporting/json_file.py`	Serializes terminal metadata.
`rampart/probes/_single_turn.py`	Clarifies turn-limit behavior.
`rampart/probes/_factory.py`	Updates probe API documentation.
`rampart/attacks/_xpia.py`	Clarifies XPIA turn limits.
`rampart/attacks/_factory.py`	Updates XPIA API documentation.
`tests/unit/core/test_types.py`	Tests enums and turn validation.
`tests/unit/core/test_result.py`	Tests result contracts and resolvers.
`tests/unit/core/test_execution.py`	Tests early trial validation.
`tests/unit/pytest_plugin/test_xdist.py`	Tests transport and truncation.
`tests/unit/pytest_plugin/test_xdist_aggregation.py`	Tests worker/controller persistence.
`tests/unit/reporting/test_json_file.py`	Tests JSON provenance output.
`docs/api/core-types.md`	Documents new core APIs.
`docs/usage/results-and-reporting.md`	Explains result provenance.
`docs/usage/xdist.md`	Documents envelope behavior.
`docs/probes/behavioral.md`	Updates probe turn-limit semantics.
`docs/attacks/xpia.md`	Updates XPIA turn-limit semantics.

Review details

Files reviewed: 22/22 changed files
Comments generated: 1
Review effort level: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

azure-pipelines · 2026-09-08T23:16:15Z

Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run.

Copilot

🟡 Changes recommended

Extreme malformed integers can escape the xdist fail-closed error boundary.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

rampart/pytest_plugin/_xdist.py:998

A sufficiently large integer overflows float(), then its !r conversion can itself raise ValueError under Python's integer-string digit limit. That exception escapes instead of becoming WorkerOutputError, so malformed worker data can abort the pytest hook rather than marking the run incomplete. Render the value through the existing safe string helper.

        msg = f"Confidence could not be converted to float: {raw_confidence!r}."

Files reviewed: 23/23 changed files
Comments generated: 3
Review effort level: Balanced

Keep numeric overflow and evaluation text conversion failures inside the worker-output error boundary so controllers mark runs incomplete and preserve earlier results.

Require explicit response scopes and distinguish online evidence from terminal verdict input without an alias. Apply canonical policies consistently to shared evaluation schemas and version the tightened population contract. Migrate callers and document direct pre-1.0 API replacement while preserving fail-closed records.

Use final_trace_evaluation consistently in Result, canonical records, JSON reports, and xdist transport. Update examples and round-trip tests without retaining an alias for the earlier spelling.

Nina Chikanov (nina-msft)

Thanks for addressing my previous comments! This round of review focused on the major bump to the schema and associated changes. I don't think the deprecation change needs a lot of discussion but is worth aligning on with Behnam at the very least (since he helped push through #185) in a quick call or in PL.

Validate inline migration instructions instead of document paths. Keep migration mechanics generic, align schema support with the project-wide deprecation policy, and focus xdist documentation on user-visible guarantees.

Nina Chikanov (nina-msft)

Thanks for the changes!

Spencer Schoenberg (spencrr) mentioned this pull request Aug 4, 2026

[FEAT]: Serialize final evaluation metadata #147

Closed

Spencer Schoenberg (spencrr) force-pushed the dev/spencrr/trace-result-contract branch from 3cc7cc8 to fb90c72 Compare August 8, 2026 02:31

Spencer Schoenberg (spencrr) force-pushed the dev/spencrr/trace-result-contract branch 2 times, most recently from c3b5f80 to 9b7c6a6 Compare August 27, 2026 17:30

Spencer Schoenberg (spencrr) force-pushed the dev/spencrr/trace-result-contract branch from 9b7c6a6 to ef98095 Compare September 8, 2026 20:41

Spencer Schoenberg (spencrr) changed the title ~~[FEAT]: Add final evaluation result contract~~ [FEAT]: Add durable terminal evaluation contract Sep 8, 2026

Spencer Schoenberg (spencrr) mentioned this pull request Sep 8, 2026

[REFACTOR]: Add shared linear trace runner #148

Open

3 tasks done

Spencer Schoenberg (spencrr) requested a balanced review from Copilot September 8, 2026 20:42

Copilot started reviewing on behalf of Spencer Schoenberg (spencrr) September 8, 2026 20:43 View session

Copilot AI reviewed Sep 8, 2026

View reviewed changes

Comment thread rampart/pytest_plugin/_xdist.py Outdated

Spencer Schoenberg (spencrr) force-pushed the dev/spencrr/trace-result-contract branch from ef98095 to acbf019 Compare September 8, 2026 22:46

Spencer Schoenberg (spencrr) marked this pull request as ready for review September 8, 2026 23:16

Spencer Schoenberg (spencrr) requested review from a team and a balanced review from Copilot September 8, 2026 23:16

Copilot started reviewing on behalf of Spencer Schoenberg (spencrr) September 8, 2026 23:16 View session

Copilot AI reviewed Sep 8, 2026

View reviewed changes

Comment thread rampart/pytest_plugin/_xdist.py Outdated

Comment thread rampart/core/_population.py Outdated

Comment thread tests/unit/core/test_types.py

Nina Chikanov (nina-msft) reviewed Sep 15, 2026

View reviewed changes

Spencer Schoenberg (spencrr) added 3 commits September 24, 2026 16:46

[FEAT]: Add durable terminal evaluation contract

cd56846

[FIX]: Contain malformed evaluation decoding failures

c8389d8

Keep numeric overflow and evaluation text conversion failures inside the worker-output error boundary so controllers mark runs incomplete and preserve earlier results.

Spencer Schoenberg (spencrr) force-pushed the dev/spencrr/trace-result-contract branch from acbf019 to 226cf4e Compare September 25, 2026 01:04

[BREAKING]: Name final trace evaluation explicitly

f98c3b2

Use final_trace_evaluation consistently in Result, canonical records, JSON reports, and xdist transport. Update examples and round-trip tests without retaining an alias for the earlier spelling.

Spencer Schoenberg (spencrr) requested a review from Nina Chikanov (nina-msft) September 25, 2026 02:32

Nina Chikanov (nina-msft) reviewed Sep 26, 2026

View reviewed changes

Comment thread schemas/trace-compatibility.json Outdated

Comment thread docs/concepts/trace-schema.md Outdated

Comment thread docs/concepts/trace-schema.md Outdated

Comment thread docs/usage/xdist.md Outdated

Comment thread schemas/trace-compatibility.json Outdated

[MAINT]: Inline trace migration notes and clarify support policy

62b9fbf

Validate inline migration instructions instead of document paths. Keep migration mechanics generic, align schema support with the project-wide deprecation policy, and focus xdist documentation on user-visible guarantees.

Nina Chikanov (nina-msft) approved these changes Sep 30, 2026

View reviewed changes

Spencer Schoenberg (spencrr) merged commit ecd2bb7 into microsoft:main Sep 30, 2026
15 checks passed

Conversation

Spencer Schoenberg (spencrr) commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Breaking changes

Checklist

Uh oh!

azure-pipelines Bot commented Aug 4, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

azure-pipelines Bot commented Sep 8, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Suppressed comments (1)

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Nina Chikanov (nina-msft) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Nina Chikanov (nina-msft) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Spencer Schoenberg (spencrr) commented Aug 4, 2026 •

edited

Loading